Skip to content

[Fix](insert) add LoadStatistic fileNumber and fileSize for insert into from s3 #57378

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Oct 29, 2025
Merged

[Fix](insert) add LoadStatistic fileNumber and fileSize for insert into from s3 #57378

merged 5 commits into from
Oct 29, 2025

Conversation

@JNSimba
Copy link
Member

@JNSimba JNSimba commented Oct 27, 2025 •
edited
Loading

What problem does this PR solve?

add LoadStatistic fileNumber and fileSize for insert into tbl select * from S3()

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 27, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@JNSimba
Copy link
Member Author

JNSimba commented Oct 28, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 28, 2025

TPC-DS: Total hot run time: 190257 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a8a750402fe9588ee45c7855a7dfa481260c3dcd, data reload: false

query1	1091	438	414	414
query2	6597	1706	1684	1684
query3	6747	224	228	224
query4	25965	23968	23037	23037
query5	5053	657	483	483
query6	354	265	237	237
query7	4652	510	310	310
query8	327	286	274	274
query9	8730	2628	2636	2628
query10	562	348	294	294
query11	15734	15138	14917	14917
query12	203	126	118	118
query13	1700	578	457	457
query14	12775	9300	9314	9300
query15	233	197	180	180
query16	7750	687	530	530
query17	1583	770	616	616
query18	2210	452	380	380
query19	411	213	184	184
query20	138	124	128	124
query21	220	155	118	118
query22	4609	4849	4528	4528
query23	34804	33994	33776	33776
query24	8267	2513	2564	2513
query25	623	533	489	489
query26	1311	290	175	175
query27	3302	551	378	378
query28	4351	2327	2301	2301
query29	801	641	516	516
query30	320	253	206	206
query31	972	904	782	782
query32	89	83	72	72
query33	643	382	342	342
query34	795	863	525	525
query35	795	830	753	753
query36	963	980	916	916
query37	117	106	85	85
query38	3567	3508	3518	3508
query39	1451	1405	1414	1405
query40	218	128	118	118
query41	65	58	62	58
query42	120	112	116	112
query43	502	502	476	476
query44	1240	752	747	747
query45	187	180	175	175
query46	891	994	645	645
query47	1786	1862	1774	1774
query48	395	420	323	323
query49	771	539	426	426
query50	645	692	412	412
query51	3951	3925	3883	3883
query52	110	112	100	100
query53	239	265	214	214
query54	598	601	556	556
query55	91	88	85	85
query56	337	325	303	303
query57	1185	1186	1140	1140
query58	294	291	274	274
query59	2624	2663	2499	2499
query60	349	341	331	331
query61	162	149	160	149
query62	797	726	676	676
query63	238	200	196	196
query64	4473	1178	848	848
query65	4057	3947	3971	3947
query66	1098	425	348	348
query67	15343	15151	14999	14999
query68	8278	933	605	605
query69	491	330	292	292
query70	1336	1337	1290	1290
query71	474	340	319	319
query72	6073	4986	4934	4934
query73	670	583	371	371
query74	9250	9160	8651	8651
query75	3799	3324	2850	2850
query76	3460	1148	735	735
query77	769	417	311	311
query78	9480	9536	8986	8986
query79	2636	848	595	595
query80	713	590	521	521
query81	528	266	227	227
query82	484	161	141	141
query83	259	270	254	254
query84	259	117	89	89
query85	930	472	451	451
query86	391	306	308	306
query87	3728	3799	3686	3686
query88	3965	2305	2302	2302
query89	403	334	308	308
query90	2010	215	212	212
query91	164	169	138	138
query92	90	69	69	69
query93	2239	993	642	642
query94	739	450	344	344
query95	404	317	317	317
query96	495	583	290	290
query97	2960	2975	2909	2909
query98	245	214	210	210
query99	1344	1442	1309	1309
Total cold run time: 282647 ms
Total hot run time: 190257 ms

@doris-robot
Copy link

doris-robot commented Oct 28, 2025

ClickBench: Total hot run time: 28.23 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a8a750402fe9588ee45c7855a7dfa481260c3dcd, data reload: false

query1	0.06	0.06	0.05
query2	0.09	0.06	0.06
query3	0.26	0.09	0.09
query4	1.61	0.12	0.11
query5	0.28	0.27	0.25
query6	1.18	0.65	0.64
query7	0.04	0.03	0.03
query8	0.07	0.04	0.04
query9	0.61	0.51	0.52
query10	0.58	0.59	0.57
query11	0.17	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.60	0.61
query14	1.03	1.03	1.01
query15	0.86	0.87	0.85
query16	0.42	0.41	0.39
query17	1.06	1.04	1.05
query18	0.22	0.20	0.20
query19	1.88	1.80	1.86
query20	0.02	0.01	0.01
query21	15.43	0.19	0.14
query22	5.10	0.07	0.05
query23	15.68	0.26	0.11
query24	2.73	0.99	1.12
query25	0.09	0.07	0.06
query26	0.14	0.13	0.13
query27	0.06	0.05	0.06
query28	5.65	1.15	0.94
query29	12.56	3.92	3.24
query30	0.29	0.13	0.15
query31	2.81	0.58	0.38
query32	3.23	0.56	0.47
query33	3.12	3.07	3.13
query34	15.91	5.15	4.54
query35	4.61	4.59	4.61
query36	0.67	0.52	0.49
query37	0.10	0.06	0.06
query38	0.07	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.16	0.15
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 99.83 s
Total hot run time: 28.23 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 28, 2025

FE UT Coverage Report

Increment line coverage 0.00% (0/19) 🎉
Increment coverage report
Complete coverage report

@JNSimba
Copy link
Member Author

JNSimba commented Oct 28, 2025

run buildall

@JNSimba JNSimba changed the title [Fix](job) add LoadStatistic fileNumber and fileSize for streaming job [Fix](insert) add LoadStatistic fileNumber and fileSize for insert into from s3 Oct 28, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 28, 2025

FE Regression Coverage Report

Increment line coverage 100.00% (15/15) 🎉
Increment coverage report
Complete coverage report

@JNSimba
Copy link
Member Author

JNSimba commented Oct 28, 2025

run performance

@doris-robot
Copy link

doris-robot commented Oct 28, 2025

ClickBench: Total hot run time: 28.29 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 03734f951a2830f388340d04af4a6dca5801edac, data reload: false

query1	0.06	0.05	0.06
query2	0.11	0.05	0.05
query3	0.26	0.08	0.08
query4	1.61	0.12	0.11
query5	0.30	0.27	0.26
query6	1.21	0.66	0.67
query7	0.04	0.04	0.03
query8	0.06	0.05	0.04
query9	0.61	0.55	0.52
query10	0.59	0.59	0.58
query11	0.17	0.12	0.12
query12	0.15	0.13	0.13
query13	0.63	0.61	0.61
query14	1.03	1.03	1.03
query15	0.90	0.84	0.86
query16	0.43	0.39	0.40
query17	1.07	1.09	1.06
query18	0.22	0.21	0.20
query19	1.97	1.82	1.87
query20	0.01	0.02	0.02
query21	15.42	0.19	0.15
query22	5.08	0.08	0.06
query23	15.66	0.26	0.12
query24	2.65	0.62	0.46
query25	0.08	0.07	0.06
query26	0.15	0.14	0.14
query27	0.07	0.06	0.06
query28	4.64	1.15	0.96
query29	12.62	4.08	3.42
query30	0.31	0.14	0.12
query31	2.82	0.61	0.39
query32	3.24	0.56	0.50
query33	3.07	3.19	3.10
query34	15.85	5.18	4.65
query35	4.63	4.61	4.64
query36	0.68	0.52	0.52
query37	0.10	0.07	0.07
query38	0.07	0.05	0.04
query39	0.04	0.03	0.04
query40	0.18	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.05	0.03
Total cold run time: 98.97 s
Total hot run time: 28.29 s

@github-actions
Copy link
Contributor

github-actions bot commented Oct 29, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 29, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 29, 2025

PR approved by anyone and no changes requested.

sollhui
sollhui approved these changes Oct 29, 2025
View reviewed changes
Copy link
Contributor

@sollhui sollhui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JNSimba JNSimba merged commit 8be5772 into apache:master Oct 29, 2025
29 of 30 checks passed
github-actions bot pushed a commit that referenced this pull request Oct 29, 2025
@JNSimba
[Fix](insert) add LoadStatistic fileNumber and fileSize for insert in…
5db46cd 
…to from s3 (#57378)

### What problem does this PR solve?

add LoadStatistic fileNumber and fileSize for insert into tbl select *
from S3()
@github-actions github-actions bot mentioned this pull request Oct 29, 2025
Merged
yiguolei pushed a commit that referenced this pull request Oct 31, 2025
@github-actions @JNSimba
branch-4.0: [Fix](insert) add LoadStatistic fileNumber and fileSize f…
f1282b1 
…or insert into from s3 #57378 (#57455)

Cherry-picked from #57378

Co-authored-by: wudi <[email protected]>
dwdwqfwe pushed a commit to dwdwqfwe/doris that referenced this pull request Oct 31, 2025
@JNSimba @dwdwqfwe
[Fix](insert) add LoadStatistic fileNumber and fileSize for insert in…
ba0fd43 
…to from s3 (apache#57378)

### What problem does this PR solve?

add LoadStatistic fileNumber and fileSize for insert into tbl select *
from S3()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@924060929 924060929 924060929 approved these changes

+1 more reviewer

@sollhui sollhui sollhui approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@JNSimba @hello-stephen @doris-robot @924060929 @sollhui @yiguolei